Added user doc for GCE HA master #1810
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
Part of kubernetes/enhancements#48 |
|
CC @kubernetes/sig-cluster-lifecycle @kubernetes/sig-cluster-ops |
|
|
||
| The sample command to set up the HA-compatible cluster: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
If you mark this as ```shell does it format nicer?
| $ MULTIZONE=true KUBE_GCE_ZONE=europe-west1-b ENABLE_ETCD_QUORUM_READS=true ./cluster/kube-up.sh | ||
| ``` | ||
|
|
||
| Please note that execution of the comments above will create a cluster with one master, |
| master. | ||
|
|
||
| * `KUBE_GCE_ZONE=zone` - zone where the master replica will run. | ||
| Should be in the same region as other replicas' zones. |
There was a problem hiding this comment.
Does the script enforce this? Or is it just recommended?
There was a problem hiding this comment.
We don't support different regions. I'll add check to kube-up script.
|
|
||
| ### Deployment best practices | ||
|
|
||
| * Try to place master replicas in different zones. During a zone failure, all master placed inside the zone will fail. |
| ### Deployment best practices | ||
|
|
||
| * Try to place master replicas in different zones. During a zone failure, all master placed inside the zone will fail. | ||
| o prevent zone failure, also place nodes in multiple zones |
| So, both replicas are needed and a failure of any replica turns cluster into majority failure state. | ||
| Such two replica setup is worse in terms of HA than a single replica setup. | ||
|
|
||
| * During addition of a master replica, cluster state (etcd) is copied to a new instance. |
There was a problem hiding this comment.
Is there a way to track the current status of the data migration to see when it completes?
There was a problem hiding this comment.
I don't know any easy way to track it. Maybe it is somehow reflected in etcd logs.
|
|
||
| When starting the second master replica, a load balancer containing the two replicas will be created | ||
| and the IP address of the first replica will be promoted to IP address of load balancer. | ||
| Similarly, when after removal of a master replica, only one replica will remain, |
There was a problem hiding this comment.
"... after removal of the penultimate master replica, the load balancer..."
|
|
||
| * Master certificates | ||
|
|
||
| Master TLS certificates will be generated for the external public IP (of the load balancer) and local IP of each replica. |
There was a problem hiding this comment.
remove "of the load balancer". I think the external ip is described sufficiently well above.
|
|
||
| Similarly, the external IP will be used by kubelets to communicate with master. | ||
|
|
||
| * Master certificates |
There was a problem hiding this comment.
I think this should be a sub-heading instead of a bullet point.
| There will be no certs for ephemeral public IP of replicas. | ||
| So, accessing them using ephemeral public IP will be possible only when skipping TLS verification. | ||
|
|
||
| * Clustering etcd |
There was a problem hiding this comment.
This should be a heading too.
|
Should you add a pointer from |
Yes, but I plan to update docs/admin/high-availability in another PR |
jszczepkowski
force-pushed
the
ha-doc
branch
2 times, most recently
from
89a58ef
to
a108a57
Compare
|
Comments applied, PTAL |
| To allow etcd clustering, ports needed to communicate between etcd instances will be opened (for inside cluster communication). | ||
| To make such deployment secure, communication between etcd instances is authorized using SSL. | ||
|
|
||
| ## Future reading |
| * `KUBE_GCE_ZONE=zone` - zone where the master replica will run. | ||
| Must be in the same region as other replicas' zones. | ||
|
|
||
| * you don't need to set `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags as they values will be inherited from already running clusters |
There was a problem hiding this comment.
This can be a paragraph instead of a bullet point, since it isn't a flag to set but rather guidance about flags not to set.
Applying LGTM |
|
I took off the lgtm because I'm not sure if we need a docs lgtm in addition to a technical lgtm (which is what the spreadsheet implies). We should clarify that and then get this merged. |
|
We need both Tech LGTM and Docs LGTM. I usually interpret the regular "lgtm" label as Tech LGTM. Doing docs review now. |
|
thanks @devin-donnelly. i've added back the lgtm label. |
|
|
||
| ## Introduction | ||
|
|
||
| In kubernetes version 1.5, we added alpha support for replication of kubernetes masters in kube-up/down scripts for GCE. |
There was a problem hiding this comment.
Avoid "we" constructs.
Suggested rephrase: "Kubernetes version 1.5 adds alpha support for replicating Kubernetes masters in kube-up or kube-down scripts for Google Container Engine."
| ## Introduction | ||
|
|
||
| In kubernetes version 1.5, we added alpha support for replication of kubernetes masters in kube-up/down scripts for GCE. | ||
| This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for GCE case. |
There was a problem hiding this comment.
"implemented for GCE case" -> "implmented for use with GCE."
|
|
||
| ## Running HA cluster on GCE | ||
|
|
||
| ### Starting HA-compatible cluster |
There was a problem hiding this comment.
Avoid nesting headings directly beneath one another; that's usually indicative of structural problems.
In this case, "Running HA Cluster on GCE" is uncessary and doesn't add anything. Move the subsequent headers one level up.
|
|
||
| ### Starting HA-compatible cluster | ||
|
|
||
| When creating a new HA cluster, two flags need to be set for kube-up script: |
There was a problem hiding this comment.
"To create a new HA cluster, you must set the following flags in your kube-up script:"
| If true, reads will be directed to leader etcd replica. | ||
| Setting this value to true is optional: reads will be more reliable but will also be slower. | ||
|
|
||
| In addition, we may specify in which GCE zone the first master replica will be created by setting: |
There was a problem hiding this comment.
"Optionally, you can specify a GCE zone where the first master replica is to be created. Set the the following flag:"
| (see [multiple-zones](http://kubernetes.io/docs/admin/multiple-zones/) for details). | ||
|
|
||
| * Do not use cluster with two master replicas. Consensus on a two replica cluster requires both replicas running when changing persistent state. | ||
| So, both replicas are needed and a failure of any replica turns cluster into majority failure state. |
There was a problem hiding this comment.
"So", "As a result,"
|
|
||
| * Do not use cluster with two master replicas. Consensus on a two replica cluster requires both replicas running when changing persistent state. | ||
| So, both replicas are needed and a failure of any replica turns cluster into majority failure state. | ||
| Such two replica setup is worse in terms of HA than a single replica setup. |
There was a problem hiding this comment.
"A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster."
| So, both replicas are needed and a failure of any replica turns cluster into majority failure state. | ||
| Such two replica setup is worse in terms of HA than a single replica setup. | ||
|
|
||
| * During addition of a master replica, cluster state (etcd) is copied to a new instance. |
There was a problem hiding this comment.
"During addition of a master replica," -> "When you add a master replica,"
|
|
||
| ### Master service & kubelets | ||
|
|
||
| Instead of trying to keep up-to-date list of kubernetes apiserver in kubernetes service, we will direct all traffic to the external IP: |
There was a problem hiding this comment.
"We" constuct. If "We" is in this case the Kubernetes system, say so.
"Instead of trying to keep an up-to-date list of Kubernetes apiserver in the Kubernetes service, the system directs all traffic to the external IP:"
|
|
||
| Master TLS certificates will be generated for the external public IP and local IP of each replica. | ||
| There will be no certs for ephemeral public IP of replicas. | ||
| So, accessing them using ephemeral public IP will be possible only when skipping TLS verification. |
There was a problem hiding this comment.
Avoid "so."
Avoid the the "them" pronoun; as written, "them" refers to "ephemeral public IPs" when it looks like you mean "master replicas. Be explicit. Also try to avoid future tense and passive voice.
Example rephrasing:
"Kubernetes generates Master TLS certificates for the external public IP and local IP for each replica. There are no certificates for the ephemeral public IP for replicas; to access a replica via its ephemeral public IP, you must skip TLS verification."
|
@devin-donnelly |
| Kubernetes version 1.5 adds alpha support for replicating Kubernetes masters in kube-up or kube-down scripts for Google Compute Engine. | ||
| This document describes how to use kube-up/down scripts to manage highly available (HA) masters and how HA masters are implemented for use with GCE. | ||
|
|
||
| ## Starting HA-compatible cluster |
There was a problem hiding this comment.
"Starting an HA-compatible cluster"
|
|
||
| ## Starting HA-compatible cluster | ||
|
|
||
| To create a new HA-compatible cluster, you must set the following flags in your kube-up script: |
| ## Adding a new master replica | ||
|
|
||
| After you have created an HA-compatible cluster, you can add master replicas to it. | ||
| You add master replicas by using a kube-up script with the following flags: |
| $ KUBE_GCE_ZONE=europe-west1-c KUBE_REPLICATE_EXISTING_MASTER=true ./cluster/kube-up.sh | ||
| ``` | ||
|
|
||
| ## Removing master replica |
There was a problem hiding this comment.
"Removing a master replica"
| * `KUBE_GCE_ZONE=zone` - zone where the master replica will run. | ||
| Must be in the same region as other replicas' zones. | ||
|
|
||
| You don't need to set the `MULTIZONE` or `ENABLE_ETCD_QUORUM_READS` flags, as those values are inherited from already running cluster, |
There was a problem hiding this comment.
Three clauses is probably one too many for this sentence. :)
"You don't need to set the MULTIZONE or ENABLE_ETCD_QUORUM_READS flags, as those are inherited from when you started your HA-compatible cluster."
|
Awesome, thanks. Just a few more changes. |
|
@devin-donnelly - have the content changed enough that i should take another pass once you are finished reviewing? |
Added user doc for GCE HA master.
|
@devin-donnelly |
|
@roberthbailey , all my comments are on doc organization and wording; the same things get said, but they may use fewer pronouns, active voice, or be said in a slightly different order. I think your LGTM should still stand. |
|
@devin-donnelly - thanks. |
|
Thanks, @jszczepkowski ! This is good to go. |
Added user doc for GCE HA master.
This change is